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PREFACE 


This  report  was  created  for  the  F-16  Aircrew  Training  De¬ 
velopment  Project  contract  no.  F02604-79-C8875  for  the  Tactical 
Air  Command  to  comply  with  the  requirements  of  CDRL  no.  8029. 

The  project  entailed  the  design  and  development  of  an  instruc¬ 
tional  system  for  the  F-16  RTU  and  instructor  pilots.  During  the 
course  of  the  project,  a  series  of  development  reports  was  issued 
describing  processes  and  products.  A  list  of  those  reports 
follows  this  page.  The  user  is  referred  to  Report  No.  34,  A 
Users  Guide  to  the  F-16  Training  Development  Reports,  for  an 
overview  and  explanation  of  the  series,  and  Report  No.  35,  F-16 
Final  Report,  for  an  overview  of  the  Instructional  System  De¬ 
velopment  Project. 
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EXECUTIVE  SUMMARY 


Within  TAC  the  complexity  of  the  weapons  systems  is  increas¬ 
ingly  rapid.  There  is  also  a  corresponding  increase  in  the 
training  requirements  for  these  systems.  In  order  to  develop  and 
maintain  an  efficient  and  effective  training  program  within  real¬ 
istic  constraints  of  manpower  and  training  resources,  the  manage¬ 
ment  system  should  be  responsive  to  the  following  needs: 

1.  Lifetime  maintenance  of  the  training  program. 

2.  standardization  of  the  approach  to  modifying  the  training 
system. 

3.  A  method  of  exploring  varying  program  alternatives. 

4.  Quick  response. 

5.  Early  detection  of  problems. 

6.  Flexibility  to  meet  changing  needs. 

7.  Self-monitoring  capabilities. 

8.  Capability  to  deal  with  large  amounts  of  data. 

9.  Establishment  of  a  historical  data  base. 

10.  Redundant  management  of  information. 

11.  Operation  of  the  system  at  the  user  level  of  training 
managers,  instructors,  and  students. 

Existing  training  management  systems  were  reviewed  and  were 
found  to  be  deficient  in  terms  of  their  capabilities  of  meeting 
the  above  needs.  For  example,  none  of  the  systems  have  provided 
adequate  support  for  the  initial  ISD  effort,  nor  were  they  found 
to  be  adequate  for  long  term  support  of  an  ISD  project. 

Based  on  the  identified  needs  and  the  information  gained  by 
examining  previous  and  existing  training  management  systems,  a 
system  is  described  which  has  the  potential  capability  of  meeting 
the  needs  of  the  F-16  training  project.  This  system  is  a  mix  of 
procedures,  personnel,  support  capabilities,  and  organizational 
structures  that  will  facilitate  the  planning,  management,  and 
implementation  of  the  F-16  program  over  the  life  cycle  of  the 
aircraft . 
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STATEMENT  OF  NEEDS 


Introduction 


Within  Tactical  Air  Command ,  the  complexity  of  the  weapons 
systems  is  increasing  rapidly.  At  the  same  time,  there  are 
constraints  on  critical  training  resources  such  as  simulators, 
flight  time,  and  qualified  instructors. 

There  is  a  critical  need  for  reliable,  systematic  methods  to 
coordinate  and  manage  the  large  number  of  training  and  resource 
variables  so  that  personnel  are  trained  as  efficiently  and  effec¬ 
tively  as  possible  under  the  existing  limitations. 

The  training  management  system  must  have  the  flexibility  to 
respond  quickly  and  effectively  to  a  wide  and  shifting  range  of 
requirements  and  situations.  This  capability  is  imperative  in 
order  to  achieve  the  high  standards  of  readiness  necessary  in  the 
face  of  accelerating  future  threat. 


Statement  of  Specific  Needs 

Eleven  specific  needs  have  been  identified  within  the 
requirement  for  an  overall  training  management  system.  These  are 
the  needs  of  a  training  program  that  continues  for  the  lifetime 
of  the  weapons  system.  They  are  listed  and  described  as  follows: 

1 .  Need  for  lifetime  maintenance  of  the  training  program 

An  ongoing  training  program  must  be  responsive  to  change  and 
revisable  so  that  quality  and  continuity  are  maintained.  For  the 
lifetime  of  the  training  program  there  will  be  changes  in  the 
operation  and  design  of  equipment,  in  personnel,  and  in  the 
overall  environment.  The  F-16  training  program  must  accommodate 
and  implement  those  changes. 

2 .  Need  to  standardize  the  approach  to  modifying  the  training 

system 

Standardization  of  procedures  is  a  necessity  in  maintaining 
an  effective  program.  In  conjunction  with  the  requirement  for 
lifetime  maintenance,  there  is  a  need  to  ensure  that  all  person¬ 
nel  adhere  to  prescribed  methods  and  practices.  Experience  has 
shown  that  idiosyncratic  changes  quickly  undermine  the  effective 
ness  of  any  training  program. 

3 .  Need  to  explore  the  effect  of  varying  program  parameters 

There  is  a  need  to  know  the  effects  of  alternative  solutions 
before  commitment  to  a  course  of  action.  In  any  phase  of  a 
training  program  unexpected  changes  may  occur  in  the  available 
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personnel,  resources,  and  time.  A  means  is  needed  to  explore  the 
effects  of  alternate  solutions  to  meet  any  new  set  of  circum¬ 
stances,  and  to  ensure  optimum  use  of  available  resources.  A 
historical  problem  in  ongoing  programs  has  been  the  inability  to 
explore  alternatives  in  depth  in  the  timeframes  available  for 
decision  making. 

4 .  Need  for  quick  response 

A  system  is  needed  that  can  respond  quickly  to  requests  for 
information  and  data  collection  and  that  can  facilitate  rapid 
revision  of  materials.  Rapid  response  time  in  a  training  mana¬ 
gement  system  can  save  considerable  time,  effort,  and  money. 

There  are  two  area 

tant .  First,  throughout  all  phases  of  training,  collection  and 
analysis  of  data  is  necessary  for  a  variety  of  reporting  and 
decision  making  purposes.  The  faster  the  system  can  adapt,  the 
less  chance  there  is  of  bottlenecking  or  poor  decision  making  due 
to  lack  of  information.  Secondly,  a  system  is  needed  that  can 
respond  quickly  in  implementing  revisions. 

5 .  Need  for  early  detection  of  problems 

Sensitivity  to  potential  and  existing  problems  is  necessary 
to  provide  optimum  management  in  all  phases  of  training.  Early 
problem  identification  is  important  in  two  ways.  It  makes  the 
problem  easier  to  solve,  and  it  minimizes  the  chance  for  program 
disruption . 

6 .  Need  for  flexibility  to  meet  changing  demands  on  the  program 

The  training  management  system  must  support  and  allow  rapid 
expansion  to  meet  increasing  threat.  It  must  also  provide 
continuous  operational  capability  within  imposed  constraints. 

Typical  problems  to  be  met  include  changing  student- 
instructor  ratios  and  shortage  of  other  critical  resources  during 
times  of  rapid  expansion.  During  times  of  constant  operating 
levels,  changing  constraints  such  as  reduction  of  available 
training  flights  and  fuel  must  be  dealt  with. 

7 .  Need  for  self-monitoring  system 

There  is  a  need  for  continually  evaluating  the  performance 
of  the  management  system  against  the  specified  operating  goals 
and  standards,  and  reporting  all  performance  problems. 

The  management  system  needs  to  constantly  evaluate  the 
accuracy  of  the  planning  data  against  available  real  world  data, 
and  to  constantly  compare  system  performance  against  operating 
goals.  There  is  also  a  need  to  screen  data  going  into  and  out  of 
the  system  to  detect  unreasonable  values. 
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There  is  a  need  to  collect,  store,  use,  and  distribute  large 
amounts  of  data.  These  data  are  generated  by,  and  required  for, 
the  operation  of  the  training  program. 

The  functioning  of  an  individualized,  competency-based 
training  program  generates  extremely  large  and  complex  data 
handling  requirements.  Without  data  handling  capabilities  it  is 
impractical  to  track  individual,  self-paced  students  and  maintain 
detailed  achievement  profiles. 

9 .  Need  to  establish  a  historical  data  base 

There  is  a  need  to  establish,  maintain,  and  apply  a  histor¬ 
ical  data  base  for  prediction  (projection),  analysis  and  documen¬ 
tation  purposes.  Typically,  each  training  problem  in  the  past 
has  been  approached  with  only  minimal  or  irrelevent  historical 
data.  The  result  is  frequent  repetition  of  mistakes,  and  the 
necessity  to  "reinvent  the  wheel".  Without  hard  data,  planning 
for  each  new  training  effort  proceeds  on  the  basis  of  personal 
recollections  and  intuitions. 


10 .  Need  for  redundancy 

There  is  a  need  to  provide  a  high  degree  of  redundancy  in 
the  training  management  system.  Redundancy  prevents  losing 
important  data  through  unforseen  circumstances.  The  system  needs 
to  include  protection  against  human  and  other  errors.  Loss  of 
critical  data  in  an  ongoing  program  greatly  reduces  effective¬ 
ness,  prevents  reliable  decision  making,  and  weakens  future 
planning  efforts. 

11 .  Need  for  the  system  to  be  operable  at  the  user  level 

It  is  important  that  the  training  management  system  be 
usable  directly  by  training  managers,  instructors  and  students. 
The  F-16  training  program  is  individualized  and  competency-based. 
There  will  be  a  continuing  need  for  all  users  to  frequently 
interact  with  the  training  management  system,  providing  and 
receiving  information.  This  interaction  must  be  direct  and  must 
not  require  extensive  training  if  it  is  to  be  practical. 
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STATEMENT  OF  REQUIRED  CAPABILITIES 


Introduction 


The  set  of  capabilities  necessary  to  meet  the  requirements 
specified  in  the  statement  of  needs  above  are  evident  in  current 
training  management  systems.  Even  though  these  capabilities 
exist,  this  study  team  has  found  that  current  capabilities  are 
deficient  in  three  important  dimensions. 

First,  existing  capabilities  have  repeatedly  proved  inade¬ 
quate  for  full  support  of  the  complex  initial  ISD  effort.  A 
variety  of  DOD  efforts  to  develop  information  and  training 
managemenht  systems  in  support  of  instructional  systems  develop¬ 
ment  (ISD)  attest  to  the  wide  perception  within  the  military  that 
existing  capabilities  in  the  area  leave  much  to  be  desired.  At 
least  a  half  dozen  efforts  in  all  branches  of  the  service  are 
working  on  this  problem  at  the  current  time. 

Second,  existing  training  managment  system  capabilities  have 
proven  inadequate  for  long  term  support  of  the  ISD  program 
throughout  the  years  following  the  initial  effort.  Needed  data 
are  ignored  or  not  collectable  within  existing  capabilities,  and 
as  a  result,  subsequent  ISD  efforts  repeat  the  mistakes  of  their 
predecessors . 

Third,  existing  capabilities  are  incomplete.  One  single 
system,  which  meets  all  of  the  needs  as  stated  above,  which  can 
provide  the  support  and  discipline  necessary  to  maintain  a  com¬ 
plex  and  sophisticated  training  system  through  years  of  changing 
weapons  system  characteristics  and  constant  personnel  rotations 
does  not  exist.  The  set  of  required  capabilities  such  a  system 
represents  is  addressed  by  isolated  special  purpose  systems, 
programs,  and  organizational  structures  which  attempt  to  deal 
with  various  pieces  of  the  problem,  but  the  set  is  incomplete  and 
the  pieces  are  usually  not  directly  applicable  to  the  F-16  needs. 
To  acquire  an  optimized  system,  then,  it  is  necessary  to  mount  a 
dedicated  effort  whose  ultimate  goal  is  to  create  from  current 
technology  an  integrated  system  based  upon  a  standardized  Air 
Force  management  system  structure. 


Statement  of  Specific  Capabilities  Needed 

The  capabilities  of  the  training  management  system  which  are 
outlined  here  respond  to  the  specific  needs  introduced  earlier. 
These  capabilities  deal  with  those  aspects  of  effective  informa¬ 
tion  handling  and  management  which  require  continual,  current, 
operational  inputs  to  system  managers. 
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1.  The  Training  Management  System  (TMS)  for  the  F-16  will 
provide  a  variety  of  capabilities  and  support  functions  which 
will  allow  timely,  ongoing  revision  and  update  of  this  training 
curriculum  and  all  attendant  data  bases.  It  will  be  self- 
documenting  as  much  as  possible  and  maintain  a  clear  record  of 
all  revision  trails.  These  features  will  enable  the  data  base  to 
be  responsive  to  required  revisions  without  loss  of  original 
quality  or  intent.  Off-the-shelf  word  processing  and  data  base 
management  system  capabilities  will  provide  basic  support 
functions.  These  will  be  implemented  within  an  environment  of 
applications  programs  in  a  time  shared  (real  time)  mode. 

2.  The  TMS  will  support  and  encourage  disciplined  and  stand¬ 
ardized  modification  of  the  data  base  for  the  lifetime  of  the 
weapons  system.  It  will  contribute  to  the  discipline  and  stand¬ 
ardization  of  the  program  through  built-in  expectations  (required 
interactions  with  the  system)  in  terms  of  data  requirements,  and 
it  will  provide  clearly  specified,  on-line  procedures,  job  aids 
and  training.  Quality  control  and  automatic  reporting  facilities 
will  help  ensure  that  standardization  and  discipline  of  revision 
and  updates  are  maintained. 

3.  The  TMS  will  allow  planners  and  decision  makers  to  explore 
the  effect  of  varying  program  parameters.  The  system  will  allow 
easy  access  to  both  historical  and  current  data  bases  in  modes 
which  allow  the  manipulation  of  critical  program  parameters  for 
simulation  (projection)  purposes  and  for  the  purpose  of  data  base 
update  and  adaptation  to  new  data.  The  system  will  support  and 
facilitate  the  generation  of  a  wide  variety  of  reports  for  plan¬ 
ning  and  management  purposes  and  provide  a  wide  range  of  ad  hoc 
computation  and  information  retrieval  functions. 

4.  This  system  will  provide  planners  and  users  with  quick 
response.  The  F-16  training  program  TMS  will  be  operating  in  a 
real  time  environment  for  certain  of  its  many  applications.  For 
these  classes  of  applications  (i.e.,  student  assignments  and 
prescriptions,  etc.)  a  real  time  response  is  essential.  Addi¬ 
tionally  the  system  will  support  a  wide  range  of  simultaneous 
activities  and  recognize  and  serve  each  of  the  many  classes  of 
users  appropriately  (students,  subject  matter  experts,  reviewers, 
authors,  managers,  administrators,  instructors,  proctors,  etc.). 

5.  The  TMS  will  provide  for  early  detection  of  problems.  The 
system  will  monitor  every  interaction  affecting  critical  program 
parameters  and  it  will  immediately  identify  deviations  from 
expected  values.  These  will  be  accompanied  by  automatic  report¬ 
ing  and  program-wide  dissemination  of  information. 

6.  A  capability  will  be  provided  which  gives  a  high  degree  of 
flexibility  to  meet  changing  demands  on  the  program.  The  F-16 
TMS  will  minimize  dependence  upon  highly  trained  people  who  will 
be  in  short  supply  during  critical  periods  of  rapid  expansion  to 
meet  emerging  threats.  The  system  will  be  greatly  expandable  or 


reducible  in  its  scope  and  capabilities,  in  a  very  short  time, 
with  a  minimum  requirement  for  specially  trained  personnel. 

7.  The  F-16  training  program  TMS  will  provide  continuous  self¬ 
monitoring  through  explicit  and  published  TMS  performance  expec¬ 
tations  which  are  continuously  and  automatically  checked.  Both 
the  expectations  and  the  reports  on  system  performance,  generated 
by  the  self-monitoring  process,  will  be  unambiguous. 

8.  The  F-16  training  program  TMS  will  handle  very  large  amounts 
of  data  in  short  periods  of  time  on  an  ongoing  basis.  It  will  be 
able  to  serve  large  numbers  and  many  different  types  of  users 
simultaneously  with  the  ability  of  very  rapid  and  extensive 
expansion  if  necessary. 

9.  The  TMS  will  establish  and  maintain  an  historical  data  base 
for  planning  and  reporting  purposes.  This  data  base  will  be 
maintained  in  a  state  which  provides  easy  access  to  all  data  for 
a  variety  of  planning,  decision  making,  auditing,  and  reporting 
purposes  as  required.  A  major  portion  of  this  capability  will  be 
supplied  by  a  highly  effective,  proven,  off-the-shelf  data  base 
management  system  (DBMS) . 

10.  The  TMS  will  provide  a  high  degree  of  reliability  and 
redundancy.  Through  the  use  of  system  incorporated  procedures, 
techniques,  and  job  aids,  and  through  the  use  of  distributed 
processing  techniques  (duplicate  hardware  at  each  training  site 
with  sufficient  communications  to  allow  remote  service  of  any 
site  from  any  other  sites)  a  degree  of  redundancy  will  be 
provided  which  will  render  the  TMS  relatively  insensitive  to 
either  rotation  of  personnel  or  malfunction  of  equipment. 

11.  The  F-16  training  program  TMS  will  be  human  engineered  for 
easy  and  effective  use  by  a  wide  variety  of  users,  ranging  from 
students  and  managers  to  high  administrators,  without  extensive 
and  specialized  training.  It  can  be  interfaced  with  at  the  user 
level  directly  by  the  user  himself. 
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EXPANDED  RATIONALE 


Introduction 


This  rationale  section  is  provided  to  further  explore  and 
explain  the  issues  central  to,  and  surrounding,  the  eleven 
specific  areas  of  needs  identified  within  the  F-16  training 
program. 

Inasmuch  as  the  F-16  effort  is  representative  of  the  growing 
trend  within  the  military  and  industry  towards  the  application  of 
sophisticated  systems  technology,  particularly  that  incorporated 
into  training  design  and  development  and  variously  identified  as 
ISD  or  Systems  Approach  to  Training  (SAT) ,  examples  will  be  drawn 
from  applicable  military  and  industrial  training  programs  using 
this  technology  to  illustrate  and  clarify  the  needs  identified. 

For  purposes  of  consistency  this  rationale  will  be  developed 
in  a  structure  parallel  to  that  of  the  Statement  of  Needs 
section.  It  will  be  augmented  by  a  support  document  (Appendix  A- 
F-16  ADP  Study,  Background  Analysis)  which  will  contain  a  more 
general  background  to  the  whole  problem  of  information  and  train¬ 
ing  management  in  ISD  and  will  represent  a  broader  perspective, 
which  addresses  the  eleven  areas  of  need  referenced  above,  among 
others . 


Specific  Rationale  for  Needs 

Eleven  specific  needs  have  been  identified  within  the 
requirement  for  an  overall  training  management  system.  These  are 
the  needs  of  a  training  program  that  continues  for  the  lifetime 
of  the  weapons  system.  They  are  discussed  with  supportive 
rationale  below. 

1 .  Need  for  lifetime  maintenance  of  the  training  program 

An  ongoing  training  program  must  be  responsive  to  change  and 
easily  revisable  so  that  quality  and  continuity  are  maintained. 
ISD  is  a  dynamic  process.  Training  for  modern  weapons  systems 
must  evolve  as  the  weapons  systems  do.  The  training  program  must 
be  kept  current  for  as  long  as  it  is  used.  This  often  requires 
ongoing  revisions  on  a  large  scale.  For  example,  in  one  recent 
ISD  effort  in  air  crew  training,  it  has  been  determined  by  the 
course  instructors  that  the  entire  syllabus  goes  through  a  major 
revision  once  every  eight  months.  This  means  that  approximately 
20%  of  the  instruction  changes  for  each  class.  The  difficulties 
of  maintaining  so  extreme  a  revision  schedule  can  easily  be 
imagined  ( EA-6B  (ICAP)  Instructional  Systems  Development  Study 
Data  Item  0003,  Contract  Number  N61339-7  5-C-0100 ,  Job  Analysis 
Document,  August  1975).  Even  in  stable  content  areas  where  there 
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are  very  infrequent  changes  to  what  is  being  taught  there  should 
be  ongoing  evaluation  of  training  effectiveness  which  may  reveal 
needed  revisions  to  how  lessons  are  being  taught.  Weak  lessons 
will  need  to  be  redone. 

This  evaluation  revision  cycle  must  continue  even  when  the 
course  has  been  tuned  up  and  judged  effective.  Training  does  not 
take  place  in  a  vacuum.  Often  students  enter  a  training  program 
from  previous  training  programs  which  contribute  to  the  basis  of 
assumptions  about  what  a  student  already  knows,  and  therefore 
about  what  he  must  now  be  taught.  Likewise  he  may  progress  from 
your  training  program  to  yet  another  follow-on  program.  Changes 
in  what  is  taught  in  any  of  these  may  require  corresponding 
revisions  within  your  program,  even  if  your  instruction  has  been 
effective . 

2 .  Need  to  standardize  the  approach  to  modifying  the  training 

system 

Standardization  of  procedures  is  a  necessity  in  maintaining 
an  effective  program.  Much  money  is  wasted  if  a  well-designed, 
systematically  developed  training  program  is  revised  and  modified 
according  to  the  personal  inclination  and  instructional  philos¬ 
ophy  of  a  succession  of  different  personnel  over  time.  In  such 
cases  the  original  ISD  investment  can  be  wiped  out  in  a  start¬ 
lingly  short  time. 

ISD  grew  out  of  a  perception  that  a  systematic,  disciplined 
approach  to  training  development  would  be  more  efficient  in 
producing  instruction,  would  be  easier  to  quality  control,  and  if 
properly  researched-based ,  would  be  more  consistently  effective 
than  the  idiosyncratic,  "artistic"  approaches  formerly  prevalent. 
A  good  ISD  approach  should  reduce  the  number  and  extent  of  revi¬ 
sions  necessary  to  obtain  effective  instruction  and  therefore 
lower  costs.  However,  because  of  the  extensive  analysis  and 
design  required  to  do  this,  most  ISD  efforts  still  represent  a 
non-trivial  investment  and  it  is  a  very  visible  investment,  being 
an  upfront  expenditure,  not  hidden  in  the  aggregate  incremental 
costs  of  extra  revisions  over  time.  This  investment  must  be 
protected  against  idiosyncratic,  "artistic"  changes. 

As  an  example  of  what  can  happen  one  might  look  at  the 
preliminary  findings  of  a  study  currently  being  funded  under 
DARPA  Contract  Number  MDA-903-77-C-0265 ,  Deliverable  Item  Number 
002AB.  Of  particular  interest  in  this  study  are  the  findings 
relating  to  the  SENSO  (Sensor  Operator)  Course.  It  was  found 
that  this  338  objective  course  had  been  totally  rewritten  from 
its  original  mediated  form  into  a  totally  lecture-based  course 
within  two  years  of  the  end  of  the  initial  ISD  development  phase. 
Further,  it  was  found  that  at  least  20%  of  those  changes  had 
resulted  in  sufficient  degradation  to  training  effectiveness  for 
the  objectives  concerned  to  generate  fleet  complaints  concerning 
the  performance  levels  of  students  graduating  from  the  program  on 


8 


these  particular  objectives.  The  conclusion  of  the  study  group 
is  essentially  that  the  entire  ISD  effort  has  effectively  been 
lost  and  must  be  redone.  This  is  by  no  means  an  isolated 
example . 

There  is  a  growing  concern  within  the  military  surrounding 
this  repeatedly  observed  degeneration  of  formally  established 
training  programs  over  time  as  personnel  rotate  and  revisions  are 
required  within  the  course.  The  term  "half-life"  of  an  ISD 
effort  is  beginning  to  be  used  occasionally  in  the  literature  to 
describe  that  period  of  time  following  the  initial  development  of 
a  training  program  until  half  of  its  objectives  have  been 
idiosyncratically  changed  according  to  the  personal  agendas  of 
personnel  rotating  through  the  activity. 

If  the  effectiveness  of  the  F-16  ISD  effort  is  to  be  main¬ 
tained  for  the  lifetime  of  the  weapons  system,  it  is  essential 
that  all  future  revisions  and  modifications  to  the  program  be  as 
systematically  developed  and  as  carefully  quality-controlled  as 
the  initial  effort.  To  maintain  consistency  of  effectiveness  and 
uniformity  of  instructional  approach,  it  is  also  essential  that 
these  ongoing  modifications  and  revisions  to  the  program  be 
carried  out  in  a  carefully  considered  manner,  utilizing  as  much 
as  possible  the  original  techniques  and  procedures  employed  in 
the  establishment  and  development  of  the  training  program.  This 
is  an  extremely  difficult  thing  to  do  in  a  "people  centered" 
system  where  procedures,  techniques  and  traditions  are  resident 
in  individual  personnel  rather  than  institutionalized  to  some 
degree  within  the  system  itself. 

3 .  Need  to  explore  the  effect  of  varying  program  parameters 

There  is  a  need  to  know  the  effects  of  alternative  solutions 
before  commitment  to  a  course  of  action.  One  of  the  most  impor¬ 
tant  aspects  of  managing  an  ISD  effort  is  planning.  The  F-16  ISD 
project  statement  of  work  reflects  this  important  emphasis. 
Throughout  the  initial  analysis,  design,  and  development  effort, 
there  are  requirements  for  extensive  advanced  planning  and 
exploration  of  alternatives  built  into  the  program.  Training 
support  requirements  analysis,  media  cost  benefit  analysis, 
generation  of  optimum  and  alternative  syllabi,  and  other  impor¬ 
tant  project  steps  and  activities  are  all  representative  of  the 
importance  placed  on  the  generation  of  good  planning  data. 

In  a  larger  sense  this  emphasis  is  indicative  of  the  highly 
analytical  approach  taken  to  training  development  within  the  Air 
Force  as  a  whole.  The  F-16  is  representative  of  modern  training 
programs  within  the  Air  Force  in  that  it  represents  the  alloca¬ 
tion  and  application  of  extremely  expensive  resources  in  very 
short  supply  in  many  cases.  It  is  essential  for  effective  plan¬ 
ning  within  the  F-16  program  to  be  able  to  extensively  explore 
the  effects  of  a  range  of  possible  decisions  at  each  critical 
decision  point  before  committing  to  expensive  courses  of  action. 
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This  is  best  accomplished  in  an  interactive  environment  where 
manipulation  of  critical  program  and  data  base  parameters  in  a 
simulation  mode  is  possible. 

In  essence,  an  environment  is  needed  whereby  decisions  can 
be  "made"  and  their  consequences  explored  by  means  of  systems 
models  before  the  decision  must  actually  be  committed  to  opera¬ 
tionally.  If  a  range  of  different  decisions  are  made,  the  conse¬ 
quences  can  then  be  compared  in  the  interest  of  ascertaining  what 
the  "best"  decision  might  be.  The  Air  Force  currently  has  avail¬ 
able  a  range  of  capabilities  in  this  general  area,  one  good 
example  being  MODIA  {Method  of  Designing  Instructional  Alterna¬ 
tives)  developed  by  RAND  Corporation.  This  type  of  modeling 
capability  should  be  available  within  the  context  of  the  training 
management  system  for  use  by  personnel  on  all  levels  of  the 
training  program,  as  an  aid  to  decision  making.  Other  available 
programs  with  similar  capabilities  include  DOSS  and  L-COM.  How¬ 
ever,  as  a  group  and  individually,  these  particular  systems  and 
subsystems  would  be  difficult  and  impractical  to  employ  in  the 
F-16  environment  on  a  continuing,  day-to-day  basis.  What  is 
needed  is  a  modeling  capability  incorporating  many  of  the  solu¬ 
tions  represented  by  the  systems  named  before,  but  one  which  is 
integral  to  the  overall  F-16  training  management  system  and  one 
which  overcomes  some  of  the  disadvantages,  especially  in  terms  of 
user  orientation  and  expertise  required,  and  also  in  terms  of  the 
hardware  required,  represented  by  the  systems  named. 

With  such  a  capability  for  exploring  the  consequences  of 
alternative  decisions,  a  wide  range  of  decisions  could  be  made 
routinely  before  each  important  decision  is  committed  to. 
Currently,  in  existing  training  management  contexts,  planners  are 
lucky  if  they  can  explore  more  than  one  or  two  alternatives,  even 
on  a  superficial  or  general  level,  before  a  decision  must  be 
made . 

4 .  Need  for  quick  response 

A  system  is  needed  that  can  respond  quickly  to  requests  for 
information  and  data  collection  and  that  can  facilitate  rapid 
revision  of  materials.  The  F-16  training  management  system  must 
meet  heavy  requirements  for  rapid  turnaround  of  information  and 
it  must  quickly  supply  appropriate  services  to  a  wide  range  of 
users.  Examples  include  a  wide  range  of  services,  ranging  from 
the  requirements  of  individual  students  whose  test  data  must  be 
appropriately  recorded  and  whose  progress  within  the  course  must 
be  facilitated,  through  immediate  generation  of  instructional 
prescriptions  and  student  directions  appropriate  to  the  student 
performance  measured  and  the  overall  progress  being  made  by  each 
student . 

Rapid  response  time  in  a  training  management  system  can  save 
considerable  time,  effort  and  money.  Given  that  the  F-16  train¬ 
ing  program  can  be  individualized,  considerable  savings  can  be 
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made  in  terms  of  training  support  and  hardware  requirements.  For 
example,  in  an  individualized  training  program,  expensive  pieces 
of  media  equipment  may  be  optimally  used  and  student  requirements 
for  these  expensive  items  can  often  be  fully  served  with  fewer 
pieces  of  equipment.  The  difference  can  be  most  easily  illus¬ 
trated  when  one  considers  the  difference  in  requirements  for  a 
large  class  of  students  proceeding  in  a  lock-  step,  group  prog¬ 
ress  model,  all  entering  a  lesson  in  the  course  which  requires 
the  use  of  equipment  such  as  slide-tape  machines.  In  this  model 
all  would  require  a  machine  at  the  same  time  and  the  total  number 
of  machines  necessary  could  conceivably  be  quite  large.  In  an 
individualized  model  however,  different  students  would  be  taking 
this  particular  lesson  at  different  times  since  some  students 
would  be  faster  than  others.  An  adaptable  training  management 
system  would  be  able  to  explore  a  variety  of  alternatives  assign¬ 
ments  for  each  student  at  each  assignment  point  in  his  progress, 
and  while  making  sure  that  full  use  was  made  of  expensive  equip¬ 
ment,  it  would  at  the  same  time  attempt  to  avoid  over schedul ing 
the  equipment  available. 

A  good  example  of  such  a  system  is  the  Advanced  Instruc¬ 
tional  System  (AIS)  for  the  Air  Force,  currently  operational  at 
Lowry  AFB .  One  of  the  characteristics  of  this  system  is  that  at 
the  time  an  assignment  is  to  be  made  to  a  student,  the  system 
quickly  ascertains  whether  or  not  needed  AV  and  other  equipment 
is  currently  available,  and  if  it  is  not,  provides  a  student  with 
an  alternative  assignment  (if  possible)  on  segments  of  instruc¬ 
tion  for  which  the  student  has  prerequisite  completed,  but  which 
do  not  require  equipment  or  resources  which  are  not  available. 

In  a  manually  operated  system  it  would  be  difficult  to  catalog, 
control  and  optimally  respond  to  the  mix  of  resource  and  student 
requirements  which  would  be  encountered,  in  a  timely  and  effi¬ 
cient  fashion. 

Another  important  area  of  quick  response  which  must  be 
served  in  the  F-16  training  program  is  the  area  of  revision  and 
updates.  The  system  must  respond  very  quickly  in  supporting/ 
implementing  ongoing  revision  and  updates  to  training  program. 
That  is,  as  deficiencies  are  detected  within  the  course,  the 
system  should  provide  as  much  support  as  possible  to  the  revision 
effort  required,  so  that  (hopefully)  the  next  group  of  students 
to  encounter  the  deficient  material  (perhaps  the  slower  students 
in  the  same  class)  would  be  given  improved  instructional  mate¬ 
rials.  The  critical  issue  in  terms  of  system  responsiveness  is 
the  simple  fact  that  as  training  programs  become  more  sophisti¬ 
cated  and  complex,  and  as  the  consequences  of  inadequate  or 
inefficient  training  become  more  expensive,  the  training  manage¬ 
ment  system  must  be  able  to  respond  with  needed  information  and 
services  in  seconds  or  minutes  rather  than  days  or  weeks. 

5 .  Need  for  early  detection  of  problems 

Sensitivity  to  potential  and  existing  problems  is  necessary 
to  provide  optimum  management  in  all  phases  of  training.  As  ISD 
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efforts  become  more  complex  and  larger  in  scope,  and  as  the  ISD 
models  used  become  more  sophisticated  and  detailed,  effective 
management  becomes  extremely  difficult. 

Two  ongoing  ISD  activities  are  offered  as  examples  of  this 
problem.  The  first  is  the  ISD  effort  to  support  the  air  crew 
training  for  the  SH-2F  LAMPS  ASW  Helicopter.  In  this  activity, 
now  in  the  final  phases  of  evaluation  in  both  the  East  and  West 
coast  training  squadrons,  almost  900  instructional  and/or 
performance  check  packages  were  developed  and  validated  in  eight 
different  media/delivery  modes  including  workbook,  computer 
assisted  instruction,  simulator  activities  and  several  still  and 
motion  audio-visual  formats.  This  effort  progressed  on  a  very 
fast  time  frame,  taking  a  little  over  two  years  to  complete  the 
entire  ISD  cycle  from  analysis  through  implementation  and  final 
evaluation.  A  large  and  constantly  shifting  ISD  team  which 
sometimes  numbered  as  many  as  fifty  people  (both  contractor  and 
Navy  personnel)  at  one  time,  and  which  involved  several  hundred 
different  individuals  over  the  course  of  the  project  was  required 
to  support  this  ambitious  and  effective  ISD  activity.  About 
thirty  major  professional  role  classifications  can  be  identified 
on  this  team. 

The  other  major  ISD  activity  to  be  used  as  an  example  is  the 
ISD  effort  for  the  P3  ORION  air  crew  training  program,  now  being 
carried  out  at  Moffett  Field.  In  this  activity  the  scope  is 
greatly  amplified.  For  example,  instead  of  900  instructional 
and/or  performance  check  packages,  more  than  6,000  are  involved. 
The  mix  of  media  are  at  least  as  complex  and  the  AV  effort  is  one 
of  the  most  ambitious  ever  undertaken  on  this  type  of  effort. 
Almost  four  years  will  be  required  to  complete  this  activity  and 
the  combined  contractor/Navy  ISD  team  will  run  consistently  more 
than  100  on-board  personnel  with  several  hundred  having  partici¬ 
pated  at  one  phase  or  another  by  the  end  of  the  project. 

When  it  is  realized  that  each  step  in  the  ISD  process 
requires  the  dedication  of  a  different  mix  of  these  personnel  and 
other  resources,  and  that  the  production  of  one  AV  package  may 
involve  nearly  100  identifiable  individual  activities  to  be 
managed,  the  true  scope  of  the  management  problem  involved  can  be 
appreciated.  For  example,  in  the  SH-2  activity  it  was  observed 
as  early  as  January  1976,  that  lesson  production  rates  were 
considerably  lower  than  those  which  had  been  projected  in  the 
planning  phases.  It  was  not  until  April  1976  that  sufficient 
data  could  be  gathered  to  isolate  and  adequately  document  the 
problem.  The  earlier  planning  figures  which  had  been  based  upon 
full-time  availability  of  subject  matter  experts  (SMEs)  were 
inadequate.  In  the  April  1976  monthly  report,  a  68.1%  level  of 
SME  time  was  documented.  (Contract  Number  N61339-76-C-055 ,  Data 
Item  Number  0007,  Monthly  Progress  Reports  for  January  to  April 
1976.)  Similarly  in  October  of  1977,  it  was  ascertained  that  the 
subject  matter  experts  available  to  the  P3  ISD  activity  were 
actually  writing  instructional  materials  only  39%  of  the  time 
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(Contract  Number  N00123-77-C-0792  Monthly  Progress  Report  for 
September  1977) . 

It  is  extremely  important  that  the  training  management 
system  be  very  sensitive  to  deviations  from  all  important  program 
parameters.  Such  deviations  should  be  treated  as  potential 
problems  and  should  be  immediately  reported  to  facilitate  early 
detection  of  potential  problems.  This  has  the  joint  effect  of 
minimizing  chances  for  program  disruption  and  maximizing  the 
probability  that  the  problem  can  be  solved  within  the  context  of 
available  time  and  resources. 

6 .  Need  for  flexibility  to  meet  changing  demands  on  the  program 

The  training  management  system  must  support  and  allow  rapid 
expansion  to  meet  increasing  threat.  It  must  also  provide 
continuous  operational  capability  within  imposed  constraints. 
Recent  build-ups  and  increased  demands  on  training  systems  such 
as  Vietnam,  graphically  illustrate  the  importance  which  should  be 
placed  on  this  critical  need. 

The  F-16  training  program  must  support  and  allow  large  scale 
and  rapid  expansion.  This  implies  that  it  should  place  minimum 
dependence  upon  resources  which  will  be  especially  scarce  at 
these  critical  times,  such  as  highly  trained  and  experienced 
personnel.  A  personnel  intensive  training  program  is  easily 
expandable  only  by  supplying  more  personnel  with  the  character¬ 
istics  required.  At  times  of  threat  these  personnel  will  be  in 
great  demand.  A  system  dependent  training  program  is  expandable 
by  increasing  system  capabilities.  These  increased  capabilities 
can  be  built  into  the  system  from  the  time  it  is  initiated  at 
very  low  cost. 

An  additional  measure  of  flexibility  is  the  ability  for  the 
program  to  provide  continuous  operational  capability  when 
critical  constraint  parameters  are  imposed.  For  example,  there 
is  a  need  for  the  training  program  to  be  able  to  quickly  and 
optimally  adjust  to  decreased  numbers  of  available  training 
resources  (shortage  of  aircraft,  shortage  of  instructor  pilots, 
shortage  of  fuel,  etc.),  and  yet  maintain  the  highest  possible 
training  effectiveness  within  the  revised  program  parameters.  An 
automated  system  can  immediately  switch  to  built-in  alternates 
with  a  minimum  of  disruption  and  required  retraining  of  program 
staff,  all  the  while  completely  documenting  and  managing  all 
aspects  of  the  change. 

7 .  Need  for  a  self-monitoring  system 

There  is  a  need  for  continually  evaluating  the  performance 
of  the  management  system  against  the  specified  operating  goals 
and  standards,  and  for  reporting  all  performance  problems  thus 
identified.  The  management  system  needs  to  constantly  evaluate 
the  accuracy  of  the  planning  data  against  available  real  world 
data  and  to  compare  system  performance  against  operating  goals. 
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There  is  also  a  need  to  screen  data  going  into  and  out  of 
the  system  to  detect  unreasonable  values.  For  example,  if  the 
training  management  system  does  incorporate  a  range  of  planning 
models  which  allow  management  personnel  to  manipulate  critical 
program  parameters,  it  should  be  anticipated  that  some  optimum 
values  of  these  parameters  will  be  incorporated  into  the 
project's  planning  expectations  once  a  decision  has  been  made. 

In  a  self-monitoring  training  management  system  these  "ideal"  or 
"optimized"  parameter  values  snould  be  continually  checked 
against  the  real  or  measured  values  of  these  parameters  in  the 
ongoing  data  base,  and  differences  should  be  reported.  If,  after 
a  body  of  data  on  the  parameter  values  has  been  gathered,  the 
observed  values  of  the  paramters  are  significantly  different  from 
the  values  assumed  in  making  important  decisions,  then  the  valid¬ 
ity  of  those  decisions  should  be  questioned  and  some  alternative 
paths  of  action  should  be  explored.  If  discrepancies  between 
assumed  and  actual  paramter  values  are  ignored  or  undetected  the 
program  could  gradually  find  itself  in  increasing  difficulty. 


Another  need  for  self-monitoring  lies  in  the  area  of  running 
simple  checks  to  detect  unreasonable  values  on  data  coming  into 
and  going  out  of  the  training  management  system.  A  straight  for¬ 
ward  example  of  this  process  (which  has  been  actually  observed) 
is  the  assignment  of  a  pilot  instructor  or  a  simulation  to  more 
sessions  on  a  given  day  then  could  possibly  be  met.  In  its 
simplest  form  this  need  consists  of  a  need  for  such  straight 
forward  (but  very  valuable)  techniques  as  eliminating  all  possi¬ 
bility  of  the  system  being  provided  data  item  "D"  when  only  "A", 
"B",  and  "C"  are  possible  choices.  If  these  errors  (probably 
representative  of  one  of  the  most  common  class  of  data  errors) 
are  detected  at  the  time  they  are  given  and  called  to  the  user's 
attention  for  correction,  a  lot  of  time  and  effort  will  be  saved 
and  considerable  expense  and  difficulty  will  be  avoided. 

8 .  Need  to  effectively  deal  with  large  amounts  of  data 

There  is  a  need  to  collect,  store,  use,  and  distribute  large 
amounts  of  data.  These  data  are  generated  by  and  required  for 
the  operation  of  the  training  program. 

The  functioning  of  an  individualized,  competency-based 
training  program  generates  extremely  large  and  complex  data 
handling  requirements.  Without  sophisticated  data  handling 
capabilities  it  becomes  impractical  to  track  individual,  self- 
paced  students  and  maintain  detailed  achievement  profiles.  It  is 
anticipated  that  the  F-16  Training  Program  will  be  an  individual¬ 
ized,  competency-based  training  program.  The  implications  of 
this  are  that  the  competencies  of  each  student  in  the  training 
program  will  be  represented  as  a  profile  of  skills  and  knowledges 
related  to  the  task  job  analysis  and  objectives  hierarchies  for 
the  F-16  pilot  tasks.  As  each  student  progresses  through  the 
training  program  developing  new  competencies,  his  competency 
profile  must  be  updated  to  reflect  the  real  state  of  his  perform- 
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ance.  It  is  anticipated  that  the  competency  profile  for  each 
student  will  be  updated  almost  hourly  throughout  the  training 
period.  It  is  also  anticipated  that  the  total  number  of  compe¬ 
tencies  constituting  the  F-16  pilot  tasks  will  number  in  the 
hundreds.  In  addition,  students  in  the  program  will  be  progres¬ 
sing  at  different  speeds  and  often  along  slightly  different  paths 
depending  upon  both  natural  ability  and  upon  the  complex  of 
skills  they  bring  to  the  training  program  from  previous  experi¬ 
ence  and/or  training. 

These  detailed  data  are  needed  both  to  more  adequately 
represent  the  performance  state  of  each  student  for  use  in 
matching  personnel  to  task  requirements,  and  for  purposes  of 
isolating  specific  training  needs,  and  also  for  purposes  of 
pinpointing  needed  improvements  and  revisions  to  the  training 
program  itself.  The  large  size  of  the  resultant  data  base,  the 
high  rate  of  information  flow,  the  frequent  update  requirements, 
and  the  necessity  to  interpret  and  report  these  data  for  a  vari¬ 
ety  of  program  related  purposes,  all  have  very  important  implica¬ 
tions  in  the  design  of  the  F-16  training  management  system.  If 
these  data  are  properly  collected  and  used,  the  result  will  be  a 
much  more  effective  and  efficient  training  program  and  a  much 
cleaner  match  of  personnel  to  job  task  requirements,  resulting 
directly  in  performance  payoff  in  the  F-16  mission. 

9 .  Need  to  establish  an  historical  data  base 

There  is  a  need  to  establish,  maintain,  and  apply  an  histor¬ 
ical  data  base  for  prediction  (projection)  analysis  and  documen¬ 
tation  purposes.  Typically,  each  training  problem  in  the  past 
has  been  approached  with  only  minimal  or  irrelevant  historical 
data.  The  result  is  frequent  repetition  of  mistakes  and  the 
necessity  to  "reinvent  the  wheel".  Without  hard  data,  planning 
for  each  new  training  effort  proceeds  on  the  basis  of  personal 
recollections  and  intuitions. 

One  important  finding  of  the  F-16  ADP  study  group  was  that 
one  of  the  most  serious  deficiencies  of  previous  ISD  efforts  has 
been  the  lack  of  consistently  collected,  detailed  historical  data 
upon  which  to  base  planning  and  decision  making  for  subsequent 
efforts.  As  ISD  models  become  more  and  more  sophisticated  and  as 
the  mix  of  personnel  and  skills  required  to  fully  implement  a 
modern  ISD  program  becomes  more  complex  the  importance  of  a 
detailed  historical  data  base  increases  rapidly. 

Two  recent  studies  serve  to  highlight  this  finding.  In  a 
report  entitled  "Analysis  of  Needs  and  Goals  for  Author  Training 
and  Production  Management  Systems,"  authored  by  Dr.  C  Victor 
Bunderson  under  DARPA  Contract  Number  MDA-903-76-C-0216 ,  it  was 
reported  that  a  survey  of  major  existing  Navy  ISD  efforts  found 
that  almost  no  detailed  historical  data  existed  which  would  allow 
the  analysis  of  the  effectiveness  or  efficiency  of  any  of  the 
hundreds  of  specific  techniques  and  procedures  utilized  or  any  of 
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the  many  different  personnel  structures  involved.  These  activi¬ 
ties  uniformly  were  unable  to  report  with  any  degree  of  certainty 
how  long  instructional  development  took  or  how  much  instructional 
development  cost  on  any  but  the  grossest  of  levels.  Typically, 
time  and  cost  determinations  were  made  in  a  post  hoc  manner  by 
dividing  the  total  amount  of  time  taken  and  the  total  amount  of 
funds  spent  by  the  number  of  instructional  packages  developed. 

No  information  was  available  on  any  detailed  sub-procedures  and 
techniques  used  in  terms  of  their  costs,  efficency,  or  effective¬ 
ness.  In  other  words  there  were  no  data  available  except  on  the 
most  intuitive  and  general  level  that  would  aid  in  planning  for  a 
follow-on  ISD  effort. 

In  this  kind  of  an  environment  it  should  be  clear  that  each 
ISD  effort  is  almost  doomed  to  repeat  the  mistakes  of  the  past, 
especially  in  an  environment  where  personnel  rotation  patterns 
almost  guarantee  there  will  not  even  be  continuity  in  terms  of 
the  people  involved  from  one  ISD  effort  to  the  next. 

More  recently  the  F-16  Aircrew  Training  Project  "Previous 
ISD  Program  Review"  prepared  by  Allen  Thompson  and  Dr.  Andrew 
Gibbons  for  the  F-16  training  program  found  the  same  problem 
exists  throughout  Air  Force  ISD  efforts,  with  isolated  excep¬ 
tions.  In  general  it  was  found  that  the  detailed  historical  data 
available  are  spotty  and  incomplete,  and  it  is  the  observation  of 
the  ADP  study  group  that  such  historical  data  as  are  available 
are  inadequate  to  use  in  planning  purposes  for  other  ISD  efforts 
attempting  to  avoid  the  mistakes  of  the  past  or  to  find  more 
efficient  procedures,  techniques  and  organizational  structures. 

Many  ISD  efforts  score  somewhat  better  on  the  utility  of 
available  historical  data  in  terms  of  student  performance 
records.  Even  here,  however,  there  is  room  for  considerable 
improvement,  in  that  often  in  the  past  student  performance 
records  were  kept  in  insufficient  detail  and  the  linkage  between 
measured  student  competencies  and  the  specific  job  task  require¬ 
ments  being  taught  on  any  detail  level  is  somewhat  obscure. 

Within  the  F-16  training  program  it  is  hoped  that  historical 
student  performance  records  can  be  kept  and  maintained  in  great 
detail,  and  that  they  can  serve  as  the  basis  for  planning  and 
decision  making  in  many  different  areas,  including  need  for  cur¬ 
riculum  revision  and  updates,  qualifications  of  student  personnel 
for  individual  assignments,  isolation  of  deficiencies  and  prere¬ 
quisite  skills  taught  by  other  training  programs,  isolation  of 
specific  training  program-related  problems  (such  as  an  instructor 
whose  students  characteristically  have  trouble  with  some  training 
related  competency  area) ,  and  so  on. 

If  the  F-16  training  program,  in  particular,  and  future  ISD 
based  training  programs  within  the  Air  Force,  in  general,  are  to 
be  optimally  served  in  the  future,  it  is  extremely  important  that 
a  detailed,  comprehensive,  constantly  maintained,  and  easily 
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manipulatable  historical  data  base  be  built-up  on  all  aspects  of 
the  F-16  training  program  from  initial  analysis  on  through 
continuation  training  for  the  lifetime  of  the  weapons  system. 
Without  this  provision  future  ISD  efforts  will  continue  to  rein¬ 
vent  training  related  wheels,  with  varying  degrees  of  success  and 
skill,  and  at  considerable  expense  in  terms  of  efficiency,  effec¬ 
tiveness  and  funds. 

ISD  efforts  should  build  on  one  another  and  avoid  the 
mistakes  of  the  past  while  continually  contributing  to  the  body 
of  knowledge,  techniques,  procedures  and  organizational  struc¬ 
tures  that  maximally  contribute  to  the  development  and  implemen¬ 
tation  of  effective  training.  It  is  important  that  this  data 
base  deal  as  much  with  how  the  training  was  developed  and  which 
procedures  and  techniques  are  most  cost  effective  and  efficient, 
as  with  the  more  commonly  collected  student  performance  data. 

10 .  Need  for  redundancy 

There  is  a  need  to  provide  a  high  degree  of  redundancy  (and 
hence  reliability)  in  the  training  management  system.  Redundancy 
prevents  losing  important  data  through  unforseen  circumstances. 
The  system  needs  to  include  protection  against  human  and  other 
errors.  Loss  of  critical  data  in  an  ongoing  program  greatly 
reduces  effectiveness,  prevents  reliable  decision  making,  and 
weakens  future  planning  efforts. 

The  F-16  training  management  system  must  incorporate  a  high 
degree  of  redundancy.  It  must  not  be  sensitive  to  personnel 
rotation  or  equipment  breaKdowns.  The  ideal  circumstance  would 
be  that  the  training  management  system  would  be  sufficiently 
standardized  and  flexible  to  allow  each  training  site  to  be 
capable  of  immediately  assuming  part  or  all  of  the  management 
load  of  any  other  training  site  in  the  event  of  unforseen  person¬ 
nel  or  equipment  loss.  In  addition,  important  records  must  be 
either  kept  at  more  than  one  site  or  they  must  be  recoverable 
from  more  than  one  local  data  base,  to  protect  against  loss  of 
key  information.  It  is  important  that  the  F-16  training  manage¬ 
ment  system  be  able  to  quickly  recover  from  a  wide  range  of 
unforseen  circumstances  without  the  loss  of  critical  information 
or  training  effectiveness,  in  a  minimum  amount  of  time. 

1 1 .  Need  for  the  system  to  be  operable  at  the  user  level 

It  is  important  that  the  training  management  system  be 
usable  directly  by  training  managers,  instructors  and  students. 
The  F-16  training  program  is  individualized  and  competency  based. 
There  will  be  a  continuing  need  for  all  users  to  frequently 
interact  with  the  training  management  system,  providing  and 
receiving  information.  This  interaction  must  be  direct  and  must 
not  require  extensive  training  if  it  is  to  be  practical.  The  ADP 
study  team  feels  strongly  that  the  training  management  system  to 
be  developed  should  be  directly  usable  by  any  of  the  many  differ- 
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ent  classes  of  users,  with  a  minimum  of  training,  and  without  the 
requirement  for  specially  trained  interface  personnel  between  the 
system  and  the  user.  One  drawback  in  terms  of  the  usefulness  for 
project  planning  purposes  of  many  of  the  sophisticated  partial 
management  capabilities  now  is  existence,  such  as  the  powerful 
MODIA  program  developed  by  RAND  Corporation  for  the  Air  Force  is 
that  the  sophisticated  system  can  be  used  directly  only  by  espe¬ 
cially  trained  teams  of  personnel  knowledgeable  with  the  use  of 
the  MODIA  system. 

This  contrasts  with  other  potential  management  system 
elements  such  as  the  TICCIT  system  utilized  for  CAI/CMI  purposes 
at  several  junior  college  and  military  installations,  and  the 
Versatile  Training  System  (VTS) ,  utilized  at  many  Marine  and 
Naval  Air  installations  as  a  personnel  qualification  system  and 
general  training  program  support  system.  An  important  character¬ 
istic  of  each  of  these  systems  is  that  they  assume  a  minimum 
level  of  systems  expertise  on  the  part  of  the  various  users  and 
each  system  contains  a  substantial  and  extensive  set  of  user 
aids,  helps  and  training  materials  on-line,  designed  to  maximize 
the  amount  of  service  that  the  casual  user  can  derive  from  the 
system  without  outside  intervention. 

The  F-16  training  management  system  will  provide  information 
to  a  wide  variety  of  users  ranging  from  secretarial  and  clerical 
personnel,  through  a  range  of  student,  instructor,  and  proctor 
personnel,  to  high  level  administrative  management  and  command 
personnel.  All  of  these  users  need  different  kinds  of  informa¬ 
tion  and  different  levels  of  detailed  reports  to  properly  fulfill 
their  functions.  It  is  important  that  this  information  be 
readily  accessible,  directly  from  the  system,  as  needed  by  each 
class  of  user,  and  that  interaction  with  the  system  be  perceived 
by  these  users  as  a  helpful  and  supportive  activity  rather  than  a 
threatening  and  punishing  exercise. 

The  training  management  system  will  be  most  successful  and 
beneficial  to  both  the  F-16  and  to  other  Air  Force  ISD  programs 
if  it  freely  and  efficiently  provides  the  information  needed  for 
decision  making,  project  monitoring  and  future  planning.  If 
information  is  difficult  to  access  and  requires  the  interface  of 
specially  trained  or  highly  skilled  systems  personnel,  the  impact 
of  the  system  will  be  lessened,  cost  effectiveness  will  be  lost, 
flexibility  will  be  diminished,  and  vulnerability  to  threat  will 
be  increased. 
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PREFERRED  SOLUTIONS 


Introduction 

This  section  provides  a  functional  description  of  the  F-16 
Training  Management  System  in  terms  of  the  capabilities  required 
to  meet  the  needs  identified.  It  will  describe  a  functional 
continuum  of  alternative  solutions  and  the  position  of  the 
preferred  solution  recommended  by  the  study  group  (on  that 
continuum) . 

The  specific  system  implications  of  each  identified  need 
will  be  outlined,  followed  by  a  summary  description  of  the  recom¬ 
mended  F-16  TMS  and  support  organizations.  Finally  a  set  of 
specific  recommendations  for  reaching  this  preferred  solution 
will  be  advanced. 


The  F-16  TMS  will  be  a  mix  of  procedures,  personnel,  support 
capabilities,  and  organizational  structures  whose  role  will  be  to 
monitor,  control,  document,  and  facilitate  the  planning,  manage¬ 
ment,  and  implementation  of  all  phases  of  the  F-16  training 
program,  for  the  lifetime  of  the  weapons  system.  All  of  the 
above  features  must  be  adequate  or  the  program  goals  can  not  be 
met.  For  example,  without  good  procedures,  no  mix  of  personnel 
or  organizational  structures  can  be  fully  effective.  Without  the 
proper  organization  there  are  severe  problems  in  effectively 
defining  lines  of  authority,  responsibility,  and  accountability. 
If  adequate  support  capabilities  are  not  available,  needed  data 
for  management  and  planning  will  be  lost  or  uncollectable ,  and 
needed  reports  and  other  information  products  cannot  be  supplied. 

While  there  are  no  good  substitutes  for  efficient  procedures 
and  organization,  to  a  certain  extent  personnel  and  support 
capabilities  can  be  manipulated  in  various  trade-offs  for  maximum 
effectiveness  in  any  given  situation. 

The  range  of  possible  solutions  to  the  F-16  training  manage¬ 
ment  problem  is  wide.  On  one  dimension  the  range  is  from  an 
organizational  structure  identical  to  those  currently  used  in 
similar  aircraft  training  programs,  to  a  totally  "revolutionary" 
or  radically  different  structure,  to  be  designed  from  scratch 
without  concern  for  existing  practice,  and  to  be  imposed  on  the 
system  from  without.  On  this  dimension  the  recommendation  of  the 
study  group  is  that  the  F-16  training  program's  organizational 
structure  should  be  an  "evolutionary"  step  up  from  current 
program  management  structures.  That  is,  cleaner  lines  of  respon¬ 
sibility  for  meeting  student  and  curriculum  performance  goals 
should  be  identified  (more  accountability)  and  accompanied  by 
increased  access  to,  and  control  of,  the  resources  necessary  to 
meet  those  goals  (more  control) . 
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On  the  dimension  of  personnel  and  support,  the  range  could 
run  from  a  fully  manual  (personnel  dependent)  system  to  an  almost 
fully  automated  (personnel  independent)  system.  The  study  group 
recommends  a  system  as  insensitive  to  personnel  turnover  as 
possible,  but  stops  short  of  advocating  a  fully  automated  TMS. 

The  reason  for  this  is  that,  while  a  growing  number  of  DOD  organ¬ 
izations  are  coming  to  recognize  that  automating  ISD  activities 
(see  Attachment  B)  is  highly  desirable,  some  activities  lend 
themselves  much  more  readily  to  automation  than  others  (See  "ISD 
Task  Priority  for  Computer  Aided  Training  Systems  Development  And 
Management" (CATSDM)  Contract  Number  N61339-77-C-0018 ,  Data  Item 
004)  . 

Some  of  the  characteristics  of  a  cost  effective  TMS  (both 
from  the  organizational  and  from  the  support  standpoints)  are 
outlined  below. 

1.  Because  of  the  need  for  lifetime  maintenance  of  the  training 
program,  there  is  a  need  for  easy  access  to  the  training  data  and 
materials  on  an  ongoing  basis  for  revision  and  update  purposes. 
The  system  implied  to  meet  this  need  would  posses  extensive  word 
processing  and  text  editing  capabilities,  as  well  as  the  facility 
to  automatically  maintain  critical  data  links  between  all  interim 
ISD  products  from  the  task/job  analysis  statements  which  define 
desired  student  competencies,  through  the  objective  hierarchy 
definitions  which  operationally  define  student  performances  to  be 
accepted  as  evidence  of  training  success.  The  system  would  have 
to  be  able  to  maintain  the  relationship  of  those  objectives  to 
one  another  within  the  syllabus,  through  the  final  specification 
documents  and  instructinal  materials  themselves.  Because  of  the 
mass  of  material  involved  and  the  complexity  of  the  relationships 
between  ISD  products,  and  because  of  the  importance  of 
maintaining  an  audit  trail  for  continuing  revision  and  documen¬ 
tation  purposes,  there  are  strong  implications  for  automating  the 
relevant  parts  of  the  training  management  system  designed  to  help 
answer  this  need. 

Organizationally  there  must  be  a  commitment  to  account¬ 
ability  and  to  maintaining  the  currency  and  completeness  of  the 
F-16  training  program,  including  both  the  finished  instructional 
materials  themselves,  and  the  supporting  ISD  documentation  and 
interim  products. 

2.  Because  of  the  pressing  need  to  standardize  the  approach  to 
modifying  the  training  system,  the  F-16  training  management 
system  must  contribute  to  the  discipline  and  standardization  of 
procedures  both  for  implementing  and  for  revising  and  modifying 
the  training  program  and  materials.  This  is  best  accomplished  by 
putting  those  procedures  as  much  as  possible  on-line  and  under 
the  control  of  the  system,  so  that  it  may  structure  the  inter¬ 
action  required  by  training  program  maintenance  and  revision 
personnel.  A  rich  mixture  of  on-line  interactive  revision 
protcols  and  job  aids,  supported  by  on-line  instructional  mater- 
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ials  for  accompl ishing  the  procedures,  will  help  ensure  that,  in 
spite  of  the  passage  of  time  and  the  rotation  of  knowledgeable 
personnel,  program  modification  and  maintenance  will  still  be 
carried  out  in  a  well  defined  and  disciplined  manner.  This  will 
serve  to  prevent  the  rapid  degeneration  of  training  effectiveness 
so  often  prevalant  when  curriculum  or  program  modification  is 
allowed  to  proceed  on  an  idiosyncratic  and  uncontrolled  basis. 

The  system  implication  is  that  a  relatively  heavy  degree  of  auto¬ 
mation  is  called  for  in  meeting  this  critical  need. 

Organizationally  the  implication  of  this  need  is  that  curri¬ 
culum  and  program  maintenance  and  modification  will  take  place 
under  the  control  of  a  centralized  organizational  structure 
responsible  for  proper  maintenance  of  the  training  program  and 
charged  with  the  maintenance  of  a  well  defined  and  disciplined 
approach  to  revision  and  update.  Further,  this  organization  must 
approach  the  problem  of  curriculum  maintenance  from  an  active, 
rather  than  passive,  standpoint.  That  is,  the  organization  must 
continually  operate  under  the  premise  that  no  training  program  is 
perfect  and  that  training  effectiveness  should  be  continually 
monitored  both  from  the  standpoint  of  meeting  the  previously 
specified  objectives  and  from  the  standpoint  of  continually 
comparing  those  objectives  to  real  world  training  requirements 
and  updating  the  training  program  accordingly. 

3.  Since  the  training  management  system  must  allow  planners  and 
decision  makers  to  explore  the  effect  of  varying  program  parame¬ 
ters,  there  is  a  strong  implication  that  both  historical  and 
current  data  bases  will  be  highly  manipulatable  on  a  real  time 
basis.  Since  these  data  bases  will  often  be  very  extensive  and 
since  the  relationship  of  critical  program  parameters  will  often 
be  extremely  complex,  real  time  simulation  of  program  performance 
and  modeling  of  alternative  program  decisions  heavily  implies  a 
relatively  sophisticated  automated  planning  support  system.  A 
variety  of  training  support  requirements  analysis  and  cost  bene¬ 
fits  analysis  programs  exist,  along  with  a  range  of  fairly 
sophisticated  general  modeling  capabilities,  which  could  be 
adapted  to  F-16  needs. 

Organizationally  the  F-16  training  program  structure  should 
be  charged  with  the  generation  and  maintenance  of  program  contin¬ 
gency  plans  and  ongoing  " state-of-the-progr am"  reports.  Addi¬ 
tionally,  the  organiztion  should  be  charged  with  the  requirement 
to  project  the  effect  of  a  range  of  alternative  decisions  before 
committing  to  any  extensive  or  long  range  plans  of  action  at 
critical  decision  points  in  the  program. 

4.  Since  there  is  a  need  for  quick  response  in  many  parts  of 
the  F-16  training  program,  a  system  is  needed  that  can  respond 
quickly  to  requests  for  information  and  data  collection,  and  that 
can  facilitate  rapid  revision  of  materials.  If  the  F-16  training 
program  is  to  be  an  individualized,  self-paced,  criterion  refer¬ 
enced  instructional  program  there  is  a  high  probability  that  the 


average  student  will  require  some  sort  of  information  service 
(test  grading,  instructional  prescription  or  assignment,  resource 
scheduling,  administrative  support,  etc.)  several  times  every 
day.  In  many  of  these  cases  the  student  will  be  unable  to 
proceed  until  the  information  is  provided.  Similarly,  other 
users  such  as  instructors,  proctors,  program  managers,  or  high 
level  administrators,  may  have  similar  needs  for  current  informa¬ 
tion  and  immediate  service.  All  of  these  classes  of  users  may 
need  to  be  served  simultaneously  and  some  users  should  be  given 
access  to  data  and  services  that  other  users  would  not  need  and 
should  not  access.  There  is  an  extremely  strong  implication  here 
for  automation  of  the  appropriate  aspects  of  the  training  manage¬ 
ment  system.  Successful  models  abound,  both  in  the  military  and 
industry,  in  the  areas  of  computer  managed  instruction,  data  base 
management,  and  general  automated  administrative  support.  The 
F-16  training  program  should  borrow  from  the  best  of  the  avail¬ 
able  models  and  augment  existing  services  and  functions  with 
capabilities  to  be  specified  by  the  ADP  planning  team  internally. 

Organizationally  the  F-16  training  program  staff  should  be 
structured  with  clearly  defined  areas  of  responsibility  and 
authority  to  facilitate  quick  and  effective  interface  with  infor¬ 
mation  and  decision-making  outputs.  In  other  words,  in  the  real 
time,  "all  at  once"  world  of  a  responsive,  automated  training 
management  system,  users  should  not  be  unduly  fettered  by  an  out¬ 
moded,  cumbersome,  and  unnecessarily  bureaucratic  management 
organization.  The  F-16  training  management  organization  should 
be  streamlined  and  oriented  toward  quick  and  effective  service 
for  all  classes  of  users. 

5.  Since  there  is  a  need  for  early  detection  of  problems,  there 
should  be  continual,  ongoing  monitoring  of  critical  program  para¬ 
meters  at  every  interaction  between  user  and  system,  supported  by 
automatic  reporting  of  any  discrepancy  between  the  value  of  those 
parameters  and  values  expected  by  the  training  management  organi¬ 
zation.  It  should  be  readily  perceived  that  such  a  degree  of 
monitoring  would  be  impractical  in  any  purely  manual,  people- 
based  system.  The  implication  is  that  a  system  which  could 
effectively  and  efficiently  meet  this  need  would  involve  a 
considerable  degree  of  automated  support. 

The  organizational  philosophy  of  the  F-16  training  manage¬ 
ment  structure  should  reflect  an  empirical  approach  where  mis¬ 
takes  and  problems  are  anticipated,  and  accepted  as  a  challenge 
and  a  focus  for  improvement,  rather  than  as  potentially  damaging 
events  which  should  be  down-played  or  concealed.  F-16  should  be 
a  model  program  for  Air  Force  ISD.  Uninhibited  publication  of 
its  problems  and  mistakes  are  every  bit  as  important  as  its 
successes.  Without  this  commitment  to  open  analysis  and  publica¬ 
tion  of  program  progress  it  would  be  all  too  easy  for  subsequent 
ISD  efforts  to  proceed  on  the  basis  of  a  set  of  false  assump¬ 
tions,  having  never  been  exposed  to  F-16  program  difficulties, 
with  the  result  that  the  cycle  of  repeated  mistakes  would  be 
continued . 
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6.  Since  there  is  a  need  for  flexibility  to  meet  changing 
demands  on  the  program,  the  F-16  training  management  system  must 
allow  rapid  redefinition  of  program  parameters,  evaluation  of 
alternative  training  conditions,  and  implementation  of  an  ongoing 
program  optimized  under  the  new  conditions.  This  means  the 
system  must  be  able  to  quickly  adapt  to  problems  like  the  sudden 
unavailability  of  critical  instructional  resources  (such  as  loss 
of  training  sorties  or  the  unavailability  of  simulators  or 
instructors) ,  by  assigning  students  automatically  to  the  most 
effective  instructional  alternatives.  In  addition,  during  times 
of  critical  threat,  the  training  management  system  must  be  able 
to  accomodate  extremely  rapid  growth  in  the  numbers  of  students, 
and  it  must  accommodate  itself  to  changes  in  program  parameters 
such  as  greatly  accelerated  training  schedules.  Since  the  most 
critical  resources  at  these  times  will  be  precisely  those  highly 
trained  personnel  who  could  best  serve  as  instructors  within  the 
course  and  as  managers  of  the  training  program,  there  is  a  need 
to  minimize  the  "people-dependence”  of  the  F-16  training  program. 
Here  again  the  implication  is  toward  automation  of  the  training 
management  and  implementation  systems.  A  well  defined  technol¬ 
ogy,  both  of  computer  managed  instruction  (CMI)  and  computer 
assisted  instruction  (CAI) ,  exists  and  is  being  effectively 
utilized  in  a  variety  of  military  and  industrial  settings.  These 
should  be  carefully  studied  and  the  best  aspects  of  each  techno¬ 
logy  should  be  incorporated,  wherever  feasible,  into  the  F-16 
training  program. 

Organizationally  the  system  must  be  supported  by  a  framework 
of  well  defined,  yet  somewhat  redundant  and  very  flexible  person¬ 
nel  roles,  where  as  much  as  possible  of  the  training  management 
load  could  be  quickly  absorbed  by  an  influx  of  new  personnel  with 
a  minimum  of  training,  supported  by  the  well  defined  procedures 
and  self-instructional  protocols  of  the  training  management 
system  itself,  and  easily  directed  by  a  relatively  small  number 
of  more  highly  trained  personnel  in  leadership  positions  within 
the  program.  To  maximize  the  efficiency  and  effectiveness  of  the 
few  highly  trained  and  experienced  personnel  who  may  be  available 
during  a  time  of  accelerated  threat,  they  should  be  supported  as 
much  as  possible  by  ongoing  monitoring  and  quality  control  func¬ 
tions  provided  by  the  automated  parts  of  the  training  management 
system. 

7.  Since  there  is  a  need  for  a  continually  self-monitoring 
system  which  provides  the  training  management  organization  with 
current  information  on  the  performance  of  the  management  system 
itself  (both  organizationally  and  otherwise) ,  there  is  again  an 
implication  that  the  F-16  training  management  system  should  have 
a  considerable  degree  of  automated  support.  An  automated  infor¬ 
mation  management  system  could  constantly  evaluate  the  accuracy 
of  planning  data  against  available  real  world  data  and  could 
compare  system  performance  against  operating  goals  at  very 
frequent  intervals.  In  addition,  an  automated  training  manage¬ 
ment  system  could  continually  screen  data  going  into  and  out  of 
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the  system  to  detect  unreasonable  values  and  instruct  interacting 
users  on  problems  encountered  before  they  go  on.  This  single 
feature  could  go  a  long  way  toward  improving  the  quality  of  data 
used  for  planning  and  decision-making  and  towards  enhancing  the 
cost  effectiveness  and  utility  of  the  system  to  all  concerned. 
Such  data  control  and  error  reduction  technology  is  well  within 
the  state  of  the  art. 

Organizationally  there  must  be  a  commitment  towards  con¬ 
tinual,  ongoing  evaluation  of  the  performance  of  both  the  train¬ 
ing  program  and  the  training  management  structure,  and  towards 
immediate  and  active  action  to  remedy  perceived  deficiencies. 

8.  The  need  to  effectively  deal  with  very  large  amounts  of  data 
both  in  terms  of  ISD  products,  program  documentation  and  training 
materials  and  in  terms  of  massive  amounts  of  student  and  program 
performance  data,  has  been  well  documented.  The  implication  of 
this  need  for  the  F-16  training  management  system  is  clearly  one 
of  automated  support.  Conventional,  off-the-shelf  data  base 
management  system  and  computer  managed  instruction  capabilities 
provide  a  very  good  starting  place  for  the  definition  of  solu¬ 
tions  to  F-16  training  program  needs. 

Organizationally  there  must  be  a  commitment  to  the  impor¬ 
tance  of  building  and  maintaining  substantial  data  bases  for 
planning  and  decision-making  purposes. 

9.  The  only  way  to  break  the  recurring  cycle  of  essentially 
starting  over  with  each  new  ISD  effort  is  to  establish  a  histor¬ 
ical  data  base  on  all  aspects  of  major  ISD  efforts.  It  is  the 
feeling  of  the  F-16  ADP  study  group  that  one  of  the  greatest 
contributions  of  the  F-16  training  program  to  Air  Force  ISD  and 
training  could  be  the  establishment  of  such  a  data  base  for  the 
first  time  on  a  level  of  detail,  and  in  a  format,  sufficiently 
manipulatable  for  the  planning  and  procurement  of  future  ISD 
efforts  in  a  more  efficient  and  effective  manner.  Again  the 
capabilities  for  the  establishment  of  such  an  historical  data 
base  exist  in  a  well  defined  information  management  and  data 
processing  capability  strongly  implying  automation  available  in 
large  part  off-the-shelf  or  through  adaptation  of  literally 
dozens  of  existing  DOD  capabilities. 

10.  The  need  for  redundancy  in  the  F-16  training  management 
system  can  easily  be  met  in  an  automated  or  semi-automated  envi¬ 
ronment  by  the  provision  of  communications  capabilities  between 
similar  hardware  configurations  at  each  training  site  sufficient 
to  allow  remote  service  of  any  site  from  any  other  site.  This  is 
a  state-of-the-art,  existing  technology,  ideally  adapted  to  the 
F-16  training  needs  and  identified  as  offering  a  high  degree  of 
reliability  and  service  to  the  F-16  training  program. 

With  well  defined,  on-line  procedures  supported  by  inter¬ 
active  protocols  and  system  supported  instruction  to  train  new 


personnel  and  with  redundant  hardware  capability,  the  F-1C  train¬ 
ing  program  will  offer  a  degree  of  reliability  and  redundancy 
unmatched  by  any  previously  training  program.  It  will  be  insen¬ 
sitive  to  either  rotation  of  personnel  or  large  scale  failure  of 
training  program  support  equipment,  while  offering  the  potential 
for  rapid,  large  scale  expansion  of  capabilities  at  minimum 
expense  and  with  maximum  effectiveness. 

11.  The  need  for  the  system  to  be  operable  at  the  user  level 
implies  that  the  user  must  be  guided  and  helped  as  much  as 
possible  at  every  interaction  with  the  system  by  the  system 
itself.  The  implication  here  is  that  an  automated  system  would 
offer  definite  advantages  in  terms  of  coaching  the  user,  screen¬ 
ing  his  data  for  errors  or  improper  assumptions,  and  in  offering 
immediate,  on-line  job  aids  or  procedures  training  as  needed. 

The  technology  of  human  engineering  protocols  used  with  automated 
systems  has  been  well  advanced  in  the  areas  of  computer  managed 
and  computer  assisted  instruction.  While  this  technology  may  not 
be  properly  or  widely  applied  in  other  areas  of  automatic  data 
processing,  a  considerable  amount  of  expertise  could  be  brought 
to  bear  in  defining  the  user/system  interfaces  for  the  F-16 
training  management  system  to  achieve  maximum  utility  and  simpli¬ 
city  for  the  user  at  any  level.  In  addition,  while  the  automated 
system  could  ensure  that  every  user  has  immediate  and  free  access 
to  the  full  range  of  data  and  information  to  which  he  is  enti¬ 
tled,  the  built  in  security  capabilities  of  an  automated  informa¬ 
tion  management  system  could  protect  against  improper  distribu¬ 
tion  of  any  information  or  data. 

Organizationally  the  implication  is  clear.  The  F-16  train¬ 
ing  management  structure  must  be  completely  committed  to  user 
service,  whether  that  user  be  a  student,  an  instructor,  a 
proctor,  a  manager,  or  a  high  level  administrator.  All  functions 
of  the  system  and  all  actions  of  the  management  structure  should 
be  aimed  toward  user  service. 
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SYSTEM  DEFINITION  SUMMARY 


The  needs  identified  by  the  F-16  ADP  Study  Group  within  the 
context  of  the  F-16  Training  Program,  and  the  preferred  solutions 
indicated  to  meet  those  needs,  imply  a  strong  degree  of  automated 
support  to  the  overall  F-16  training  management  system.  Summa¬ 
rizing  some  of  the  most  important  of  those  implications  for  auto¬ 
mation,  the  F-16  training  management  system  ADP  support  hardware 
should  be  time-shared  and  operating  in  an  interactive  real  time 
mode.  There  should  be  an  efficient  terminal  distribution  net  and 
both  local  and  remote  communications  capabilities  to  remote  ter¬ 
minals  and  to  other  F-16  training  management  system  computers. 

The  hardware  should  be  easily  and  effectively  upward  expandable. 
That  is,  the  system  configuration  selected  to  meet  F-16  needs 
should  not  be  the  "top  of  the  line"  of  its  hardware  type,  unable 
to  effectively  accommodate  further  growth  and  increased  require¬ 
ments.  The  hardware  system  as  installed  should  include  consider¬ 
able  excess  capacity  over  that  anticipated  for  the  initial  utili¬ 
zation.  This  excess  capacity  should  be  present  both  in  terms  of 
computer  power  and  in  terms  of  disk  and  core  memory.  These  are 
extremely  important,  and  in  the  long  run,  very  cost  effective 
considerations  in  dealing  with  the  important  needs  for  flexi¬ 
bility,  reliability  and  redundancy.  The  system  should  be  grace¬ 
fully  degradable  in  both  hardware  and  software  to  help  meet  these 
same  ends  and  increase  the  probability  that  effective  TMS  support 
can  continue  under  unusal  circumstances,  including  hardware  prob¬ 
lems.  Also  the  hardware  configuration  selected  should  be  care¬ 
fully  chosen  with  an  eye  toward  maintainability  to  reduce  the 
expense  of  the  required  spares  inventory  and  in  keeping  with  the 
redundancy  and  reliability  philosophies.  As  much  as  possible 
there  should  be  interchangeable  parts  between  duplicate  and 
similar  units  and  access  in  a  timely  manner  to  a  high  level  of 
maintenance  services.  Finally  the  hardware  system  configuration 
must  be  simple  and  reliable  enough  to  operate  and  support  with  a 
minimum  of  full  time  operating  and  support  staff  (preferably 
none)  . 


From  a  software  standpoint  the  system  selected  should  incor¬ 
porate  advanced  capabilities  in  word  processing,  text  editing, 
and  both  assembly  and  high  level  language  capabilities.  In  addi¬ 
tion  there  must  be  available  sophisticated  data  base  management 
systems,  disk  operating  systems  and  time  sharing  executives  in 
order  to  fill  the  needs  identified.  Also  from  the  software 
standpoint  there  must  be  good  data  protection  and  crash  recovery 
characteristics,  good  data  logging  capabilities,  and  in  general, 
full  capability  to  support  and  maintain  a  reliable  multi-user, 
multi-function  environment.  Software  support  characteristics  of 
the  system  should  make  it  possible  to  implement  a  sophisticated 
modeling  capability  either  through  existing  or  specially  created 
applications  programs.  It  must  be  possible  to  construct  and 
maintain  extensive  and  sophisticated  interactive  protocols  and 
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user  training  and  input  sequences  to  ensure  that  the  system  is 
fully  usable  by  the  full  range  of  intended  users  in  a  rewarding 
and  non- threatening  interaction  climate.  And  finally,  because  of 
the  intended  multi-user,  multi-function  operating  context,  it  is 
extremely  important  that  the  system  embody  good  data  protection 
and  user  security  techniques  to  avoid  the  loss  of  critical  data 
or  the  accidental  or  intentional  destruction  of  important 
records . 
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ORGANIZATION  DEFINITION  SUMMARY 


The  degree  of  automation  implied  by  the  needs  and  preferred 
solutions  identified  by  the  F-16  ADP  study  group  for  the  F-16 
training  management  system  represents  a  considerable  step  forward 
from  most  existing  training  management  systems.  For  this  reason 
it  may  excite  considerable  attention  and  interest  both  within  and 
without  the  program.  Equally  if  not  more  important  than  the 
procedural  innovations  implied  by  this  level  of  automation,  are 
the  implications  for  organizational  tasking  and  structure  implied 
by  those  needs.  The  most  advanced  automated  training  management 
system  within  the  state  of  the  art  today  would  still  be  insuffi¬ 
cient  to  meet  the  full  training  management  needs  for  a  sophisti¬ 
cated  individualized  self-paced  competency  based  training  program 
such  as  F-16,  unless  that  program  were  being  implemented  within 
an  organizational  structure  whose  aims,  tasking,  and  manpower 
make-up  were  properly  defined  to  fully  meet  all  the  goals  of  an 
advanced  ISD  program,  while  taking  full  advantage  of  the 
increased  capabilities  inherent  in  the  degree  of  automated 
support  provided. 

The  F-16  training  management  organization  should  be  tasked 
with  a  high  degree  of  accountability  for  the  on-going  maintenance 
and  revision  of  the  training  program  in  terms  of  maintaining  both 
its  currency  and  its  effectiveness.  This  responsibility  should 
be  centralized  and  accompanied  by  commensurate  control  of 
resources  adequate  to  do  the  job.  The  organization  should  be 
tasked  with  actively  pursuing  on-going  curriculum  evaluation  and 
also  for  the  on-going  evaluation  of  the  effectiveness  of  the 
supporting  training  management  organization  (i.e.,  itself).  A 
major  responsibility  of  this  organization  should  be  to  publish 
all  findings  both  positive  and  negative  in  terms  of  effectiveness 
of  the  training  program  and  the  effectiveness  of  the  management 
system  and  organization.  Without  this  kind  of  openness,  an 
on-going  commitment  to  clear  isolation  of  problem  areas,  ISD  will 
continue  to  essentially  start  afresh  from  the  same  base  of  ignor¬ 
ance  in  each  new  major  iteration  in  the  future. 

The  training  organization  must  be  responsible  for  generating 
a  variety  of  training  contingency  plans  to  include  in  part, 
alternative  plans  dealing  with  such  problems  as  sudden  increases 
in  the  number  of  students,  sudden  decreases  in  the  amount  of 
training  time  available  per  class,  loss  of  simulators  or  other 
critical  equipment,  decreases  in  the  number  of  available  training 
sorties,  decreases  in  the  number  of  trained  instructors,  etc. 
Within  the  training  management  organization,  there  must  be 
clearly  defined  lines  of  authority  and  responsibility  and  the 
primary  design  parameter  in  establishing  those  lines  should  be 
the  paramount  requirements  that,  in  effectively  implementing  any 
sophisticated  ISD  programs,  all  the  right  things  must  be  done 
(i.e.,  do  not  leave  out  critical  activities),  those  things  must 
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be  done  right  (i.e.,  lip  service  to  any  critical  activity  may  be 
more  damaging  than  ignoring  it  altogether)  ,  and  user  service  must 
be  facilitated  (i.e.,  the  training  management  organization  exists 
to  serve  the  many  different  levels  of  users,  not  the  other  way 
around) .  The  training  management  organization  must  be  committed 
to  building  and  maintaining  a  detailed,  comprehensive,  historical 
data  base  for  planning  and  projection  purposes.  It  must  be 
completely  user  oriented,  and  built  into  the  training  management 
program  should  be  recurrent  user  evaluations  of  the  training 
system  and  organization  from  all  levels  of  users. 


t 


SUMMARY  AND  RECOMMENDATIONS 


Some  preliminary  conclusions  derived  from  the  study  group 
are  summarized  below.  Most  of  these  came  directly  from  the  needs 
analysis  in  the  preceding  sections. 


Summary  Points 

1.  All  phases  and  principal  activities  of  the  ISD  process  can 
benefit  from  some  degree  of  automated  support.  (See  "ISD  Task 
Priorities  for  CATSDM"  Contract  Number  N61339-77-C-0018 ,  Data 
Item  004  and  "List  of  Candidate  ISD  Task  Processes  or  Products" 
Contract  Number  N61339-77-C-0018 ,  Data  Item  001) 

2.  Automated  support  of  ISD  for  a  rapidly  changing  weapons 
system  is  particularly  important.  This  support  maintains  the 
currency  (external  validity  with  the  real  world)  and  consistency 
(internal  validity  and  agreement  of  all  parts)  of  the  training 
program. 

3.  Increased  efficiency  and  effectiveness  of  the  F-16  training 
program  which  results  in  even  a  fraction  of  one  percent  opera¬ 
tional  improvement  will  more  than  pay  for  even  the  most  elaborate 
information  management  system  being  envisioned. 

4.  Even  partial  applications  of  the  type  of  system  being 
suggested  will  produce  an  observable  effect  in  efficiency  and 
effectiveness . 

5.  The  complete  information  management  needs  of  F-16  are  not 
known  at  this  time.  Better  functional  specifications  are  needed 
to  facilitate  Air  Force  decision  making.  The  specification 
process  should  be  ongoing  throughout  Phases  IV  and  v  of  the  ISD 
project,  at  least. 

6.  A  minimum  level  of  effort  required  to  generate  the  needed 
level  of  primary  specification  in  a  useful  time  frame  would  be 
about  3  man  years  of  effort  as  early  in  Phase  IV  as  possible. 

7.  For  those  needs  which  are  known,  there  are  existing  systems 
which  either  address  the  need  or  which  contain  adaptable  solu¬ 
tions  to  similar  problems.  (See  "Analysis  of  Existing  Programs 
for  CATSDM",  Contract  Number  61339-77-C-0018,  Data  Item  005). 

8.  No  existing  system  or  useful  combination  of  systems  add¬ 
resses  all  of  the  currently  identified  F-16  ISD  needs.  (See 
Point  7  above) . 

9.  Useful,  lew  risk,  relatively  inexpensive  (although  incom¬ 
plete)  alternatives  exist.  These  could  be  pursued  either  singly 


or  in  combinations  while  the  Air  Force  decision  making  process 
proceeds.  (See  Recommendations  Section.) 

10.  Failure  to  initiate  interim  activity  such  as  a)  the  specifi¬ 
cation  activity  in  Phase  IV,  b)  some  interim  development  and 
production  managment  support,  and  c)  word  processing  approaches 
will  result  in  a  loss  of  full  capability  in  the  eventual  inte¬ 
grated  F-16  system.  This  loss  of  capability  is  due  to  the  rapid 
pace  of  the  project  activities  and  the  lead  times  required  for 
many  functions. 

11.  Potential  benefits  to  F-16  of  a  comprehensive,  integrated 
information  management  system  include: 

—  Detailed  planning  support  of  all  levels. 

—  Identification  of  inefficiencies  and  potential 
problems . 

—  Optimizing  use  of  equipment  and  assets. 

—  Reduced  life  cycle  costs. 

—  Increased  life  cycle  efficiency  and  effectiveness. 

—  Life  cycle  contribution  to  program  discipline, 
continuity,  and  consistency. 

--  Increased  flexibility  and  responsiveness  to  meet 
threat  or  changing  conditions. 

—  Improved  information  flow  with  increased  management 
visibil ity . 

—  The  resulting  historical  data  base  will  free  sub¬ 
sequent  programs  from  repeating  the  mistakes  of  the 
past  and  can  guide  future  procurements. 

12.  The  optimal  TMS  has  profound  implications  for  other  weapons 
systems . 

13.  To  optimize  the  chances  for  successfully  meeting  the  TMS 
needs  of  the  F-16  program  anticipated  in  the  original  statement 
of  work  and  verified  and  further  defined  by  the  study  group, 
existing  systems  such  as  the  Navy's  Versatile  Training  System 
(VTS)  and  the  Air  Force's  Air  Force  Operations  Resource  Managment 
System  (AFORMS)  should  be  viewed  as  foundations  upon  which  to 
build  the  F-16  TMS.  (See  "Versatile  Training  System  (VTS)  Func¬ 
tional  Description  (FD)  For  Naval  Aviation  Activities,  REG 
31408-88-76,  15  April  1977,  Naval  Weapons  Center,  Code  31408, 
China  Lake,  California  and  Air  Force  Operations  Resource  Manage¬ 
ment  System  (AFORMS),  Functional  Description,  1  December  1976). 
Recommendations 
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1.  Word  processing,  tentatively  approved  for  Phase  IV  should  be 
maintained  for  the  rest  of  the  project  and  expanded  to  support 
production/revision. 

2.  While  decision/approvals  are  being  made,  support  for  these 
ADP  related  activities  during  Phase  IV  is  essential: 

—  Word  processing. 

--  CAI/CMI  source  study. 

--  Install/adapt  some  level  of  automated  management  support 
in  Phase  IV  for  use  in  supporting  the  large  scale, 
complex  development  and  production  events  of  Phase  V. 
(Recommend  Author  Management  System  (AMS) ,  developed  for 
DOD  under  DARPA  Contract  Number  MDA  903-76-C-0216) . 

—  Automate  task  and  goal  analysis  (should  be  integrated 
with  the  system  above  to  start  with) . 

—  Fully  analyze  existing  capabilities  (VTS  and  AFORMS 

recommended),  write  specifications  for  additional  needed 
TMS  modules,  and  contribute  other  information  needed  to 
support  specific  Air  Force  decision  making  at  each  phase 
(at  least  3  man  year  effort  in  Phase  IV) . 

3.  The  resulting  F-16  TMS  system  should  acquire  and  control  its 
own  dedicated  hardware  since  the  degree  of  interaction  required 
and  the  amounts  of  data  to  be  generated  are  too  large  to  allow 
any  chance  of  less  than  full-time,  real-time  service.  If  the 
acquisition  of  dedicated  hardware  and  use  of  a  time  shared  system 
is  necessary,  then  the  TMS  should  be  guaranteed  highest  priority. 

4.  For  reliability,  redundancy,  and  long  range  economy  purposes 
a  "multi-minicomputer"  with  local  terminal  nets  hardware  approach 
replicated  at  each  training  base  is  recommended  over  possible 
alternative  approaches  such  as  distributing  all  services  to  all 
bases  from  one  large  central  system. 
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